Method and device for image and video transmission over low-bandwidth and high-latency transmission channels

ABSTRACT

A method for transmission of images and/or video over bandwidth limited transmission channels having varying available bandwidth, which uses a classification algorithm to decompose the images and/or video to be transmitted into multiple spatial areas, each area having a specific image type; detecting the image type of each of those areas; and separately selecting for each of those areas an image and/or video encoding algorithm having a compression ratio. The classification algorithm is used to prioritize each of the areas, increase the compression ratio of the image and/or video encoding algorithm dedicated to spatial areas having lower priority in case of decreasing bandwidth.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to systems and methods for transmission ofhigh quality images and video over (wireless) bandwidth limited, lossyand high-latency transmission channels. With “high quality images andvideo” is typically meant “medical images and video”. More particularly,the invention relates to systems and methods for transmitting images andvideo over channels such as Wifi, UWB, Bluetooth, DECT, ZigBee, . . .and provides solutions for typical problems such as overall low imagequality at higher compression ratio and high latency.

BACKGROUND OF THE INVENTION

This invention is in the field of methods and systems for transmittingimages or video over channels where there is a bandwidth limitationpresent, where the channel possibly is lossy or where latency isimportant. The general problem to solve is the transport of high-qualityimages or video from one point to another. Because of the bandwidthlimitations of the channel used it may be necessary to tolerate somedecrease in image or video quality, this is for instance the case whenthe image or video data is compressed in a lossy way before sending itover the channel. In that case the decompressed image or video sequencewill be of lower quality than the original image or video sequence.Frequently used compression schemes are JPEG, JPEG2000, MPEG-2, . . . .Each of these compression schemes has its advantages and disadvantagesrelated to compression ratio, complexity of the scheme, calculationpower needed, introduced latency, . . . and the best compression schemeto use also typically depends on the contents of the image or video thatis compressed. For example: a medical image would require anothercompression scheme for optimal quality and compression ratio than aclip-art picture. Especially for demanding applications such as medicalimaging a general decrease in image quality might not be acceptable sothe bandwidth limitations clearly are in conflict with the need for highquality of at least the medical image being transmitted.

In case the transmission channel has a significant bit-error-rate (ifthe channel introduces errors in the transmitted data) then thedegradation of the image or video quality might be totally unacceptable.Especially the combination of bit errors and compressed data typicallyintroduces large errors in the decompressed data stream. This problem isreferred to as error resilience. One possible solution in case ofchannels with high bit-error-rate is to use error detection codes orerror correcting codes (ECC). These codes allow for detecting and evencorrecting bit errors within certain limitations. Examples of errorcorrecting codes are CRC (cyclic redundancy codes) and RS (Reed-Solomoncodes).

Apart from possible quality degradation compression also adds some extralatency because it takes time to compress and to decompress the stream.In some applications latency is a big problem. This is in particular thecase if the data traffic is bi-directional. This is for example the caseif a server generates video sequences, this video data is transmitted tothe client, and the client has the possibility to interact with theserver (by moving a mouse pointer for instance). In that case a highlatency will immediately be visible as slow response of the system: itwill take an unacceptable long time before the result of an action(mouse movement for example) of the client will take effect.

Furthermore, the device that receives the image or video data willtypically be a portable device and therefore will have a wirelessinterface to the server. This aspect of portability results in extracomplication what concerns power consumption, signal integrity (varyingsignal quality and possible loss of connection with the server) andsecurity aspects. Indeed, portable devices are normally battery poweredand to reach a useful battery operation time it is required to minimizethe power consumption as much as possible. This conflicts with thebandwidth limitations of typical transmission channels (Wifi802.11.a/b/g/ . . . , DECT, ZigBee, UWB, . . . ): because of the lowavailable bandwidth a high compression ratio is needed and thistypically requires a complex compression scheme. The complexity of thecompression scheme is directly related to power consumption, as extracalculations require additional power. Portable devices also suffer fromvarying signal quality. With Wifi for instance, moving the device only afew meters can result in only half the available bandwidth because ofcomplex reflections and inference of the wireless signal. Worst case,the signal can even fade away completely (for a limited time). In such asituation it is of course not desired that the portable device cannot beused at that moment. Because the data is transmitted wirelessly it iseasier for other people to receive the data. In case the data isconfidential (such as medical images or patient data) this will needextra precautions such as encryption algorithms.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a method and deviceto solve problems that currently exist with transmission of image- andvideo transmission over bandwidth-limited networks, in particularhigh-quality or medical image- and video transmission. Such problems aree.g., but not limited thereto, low overall image/video quality when onlya low bandwidth channel is available, severe image degradation due tointroduction of bit errors on the channel and the problem of latencywhen communication is bidirectional. The present invention will disclosea solution based on automatic selection of the best codec type dependingon spatial location in the image, methods to improve security andpriority signalling in a multi-user environment, methods to make maximumuse of available calculation power by automatically reconfiguringcalculation blocks, methods to reduce and hide high latency problems,and methods to reduce power consumption for portable battery-operateddevices.

In a first aspect, the present invention provides a method fortransmission of images and/or video, in particular e.g. high qualityimages and/or video, over bandwidth limited transmission channels havingvarying available bandwidth, between a server and multiple devices. With“high quality images and/or video” is typically meant “medical imagesand/or video”. Available bandwidth may be varying due to the intrinsicbandwidth being varying e.g. in case of wireless transmission, or, incase of a fixed bandwidth being present, due to variable load orvariable throughput. The method according to the present inventioncomprises the use of a classification algorithm for each of the imagesand/or video to be provided to a device, for:

-   -   decomposing the images and/or video to be transmitted into        multiple spatial areas, each area having a specific image type;    -   detecting the image type of each of those areas    -   separately selecting for each of those areas an image and/or        video encoding algorithm having a compression ratio.

According to the present invention, each of the devices are prioritized,and the classification algorithm increases the compression ratio of theimage and/or video encoding algorithms dedicated to a device havinglower priority in case of decreasing bandwidth.

According to embodiments of the present invention, the prioritizing ofthe devices may be done based on the applications accessed through eachof the devices. Alternatively, according to embodiments of the presentinvention, the prioritizing of the devices may be done based on theidentity of users using said devices, e.g. whether the user is a doctoror an assistant. The method then may include a step of user log-on toone of the devices. Alternatively, according to embodiments of thepresent invention, the prioritizing of the devices may be done based onlocation of the devices, e.g. whether a device is in an emergency roomor in a consultation room.

The method may be used in a hospital environment.

In a second aspect, the present invention provides a method for securingtransmission of data from a server to a portable imaging device, themethod comprising:

-   -   determining the exact position of the portable imaging device        with respect to an authorised area,    -   based on determined the exact position of the portable imaging        device, determining whether the portable device is authorized to        receive specific data over a predetermined transmission channel,    -   transmitting, from the server to the portable imaging device,        the specific data requested if authorisation is granted, the        portable imaging device having a display area.        The method furthermore comprises the step of removing at least        from the display area at least confidential data when the        portable imaging device leaves the authorised area.

According to embodiments of the present invention, the methodfurthermore may be adapted for showing at least on the display area atleast confidential data when the portable imaging device enters theauthorised area. According to embodiments of the present invention, themethod may furthermore be adapted for removing at least confidentialdata from volatile and/or non-volatile memory elements within theportable imaging device when the portable imaging device leaves theauthorised area.

Possibly, according to embodiments of the present invention, the methodmay furthermore comprise encrypting confidential data when the portableimaging device leaves the authorised area. According to embodiments ofthe present invention, the method may comprise a step of decryptingconfidential data when the portable imaging device enters the authorisedarea. According to embodiments of the present invention, the method mayfurthermore comprise the using of a different transmission channel fortransmitting the requested data, the transmission channel used dependingon the determined the exact position of the portable imaging device.According to embodiments of the present invention, the portable imagingdevice may determine which transmission channel to use for transmittingthe requested data. Alternatively, the server may determine whichtransmission channel to use for transmitting the requested data.

The method may be used in a hospital environment.

In a third aspect, the present invention provides a method for reducinglatency in a client-server computer system, the server being adapted forgenerating data at least dependent on one or more parameter values, themethod comprising the steps of:

-   -   predicting possible reachable future parameter values,        predicting possible future parameter values is performed by the        client, after which these predicted parameter values are sent to        the server;    -   generating data corresponding to the predicted parameter values,        and sending this data to the client, and    -   the client caching this generated data corresponding to        parameter values for future use.

According to embodiments of the present invention, the client may usethe cached data when a corresponding parameter value is set. Accordingto embodiments of the present invention, the client may use the cacheddata when a parameter value is set which falls within a predeterminedrange around the parameter valued for which the cached data had beengenerated.

The method may be used in a hospital environment.

In a fourth aspect, the present invention provides a method fortransmission of images and/or video, in particular e.g. high qualityimages and/or video, over bandwidth limited transmission channels havingvarying available bandwidth. With “high quality images and/or video” istypically meant “medical images and/or video”. Available bandwidth maybe varying due to the intrinsic bandwidth being varying e.g. in case ofwireless transmission, or, in case of a fixed bandwidth being present,due to variable load or variable throughput. The method comprises theuse of a classification algorithm for

-   -   decomposing the images and/or video to be transmitted into        multiple spatial areas, each area having a specific image type;    -   detecting the image type of each of those areas;    -   separately selecting for each of those areas an image and/or        video encoding algorithm having a compression ratio.

According to the present invention, the classification algorithm isadapted to prioritize each of the areas, the classification algorithmincreasing the compression ratio of the image and/or video encodingalgorithm dedicated to spatial areas having lower priority in case ofdecreasing bandwidth.

According to embodiments of the present invention, the method may beused in a hospital environment.

In a fifth aspect, the present invention provides a method fortransmission of images and/or video, e.g. high quality images and/orvideo, over a transmission channels from a server to a client. With“high quality images and/or video” is typically meant “medical imagesand/or video”. The method comprises the steps of

-   -   decomposing the images and/or video to be transmitted into        multiple spatial areas, each area having a specific image type;    -   detecting the image type of each of those areas;    -   separately selecting for each of those areas an image and/or        video encoding algorithm using a code for encoding the images        and/or video of the area. According to the embodiments of the        invention, the client is a reconfigurable device. The method        further comprises the step of reconfiguring this reconfigurable        device for decoding the images and/or video of the areas.

According to embodiments of the present invention, the method mayfurther comprise the steps of

-   -   adaptation of the encoding algorithms used for the encoding, the        adaptation being based on current or predicted transmission        channel properties;    -   reconfiguring the reconfigurable device for decoding the images        and/or video of the areas, based on the adapted image and/or        video encoding algorithms.

According to embodiments of the present invention, all used image and/orvideo encoding algorithms may be available at the reconfigurable device.Alternatively, according to embodiments of the present invention, onlypart of the image and/or video encoding algorithms may be available atthe reconfigurable device, whereas the method further comprises the stepof downloading image and/or video encoding algorithms not beingavailable at the reconfigurable device. According to embodiments of thepresent invention, the downloaded image and/or video encoding algorithmsmay be saved at the reconfigurable device.

According to embodiments of the present invention, the image and/orvideo encoding algorithms to be downloaded may be sent over a separateconnection between server and reconfigurable device.

According to embodiments of the present invention, the reconfigurationmay be a partial reconfiguration of the reconfigurable device. Thereconfiguration may be done from server.

The method may be used in a hospital environment.

In a sixth aspect; the present invention provides a method fortransmission of images and/or video, e.g. high quality images and/orvideo over bandwidth limited transmission channels having varyingavailable bandwidth. With “high quality images and/or video” istypically meant “medical images and/or video”. Available bandwidth maybe varying due to the intrinsic bandwidth being varying e.g. in case ofwireless transmission, or, in case of a fixed bandwidth being present,due to variable load or variable throughput. The method comprises theuse of a classification algorithm for

-   -   decomposing the images and/or video to be transmitted into        multiple spatial areas, each area having a specific image type;    -   detecting the image type of each of those areas    -   separately selecting for each of those areas an image and/or        video encoding algorithm having a compression ratio.        The method further comprises the steps of    -   encoding each of the areas by an image and/or video encoding        algorithm    -   transmitting the encoded images and/or video;    -   decoding each of the areas by an image and/or video encoding        algorithm;        wherein prior to encoding at least one of the area is provided        with padding pixels, which padding pixels are replaced by part        of one of the other areas during decoding.

According to embodiments of the present invention, the padding pixelsrepresent zones where at least two areas overlap.

The method may be used in a hospital environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically shows a desktop comprising an image and/or videofor transmission over bandwidth limited transmission channels havingvarying available bandwidth.

FIG. 2 schematically illustrates the use of a classification algorithmfor examination of a total area of an image and/or video fortransmission over bandwidth limited transmission channels having varyingavailable bandwidth.

FIG. 3 schematically shows two areas overlapping in an image and/orvideo for transmission over bandwidth limited transmission channelshaving varying available bandwidth.

FIG. 4 schematically shows a hospital environment wherein a number ofsecurity levels are defined.

FIG. 5 schematically shows a hospital environment wherein a number oftransition zones are defined

DETAILED DESCRIPTION

According to a first aspect of the present invention, a method andsystem are disclosed that allow for higher overall image qualitycompared to existing methods, when only a limited bandwidth isavailable. This is achieved by decomposing the image/video to betransmitted into multiple spatial areas, each having a specific imagetype, and detecting the image type of each of those areas and selectingthe optimal image/video codec for those areas. Selecting the optimalimage codec could mean optimizing on image quality (PSNR, MSE),optimising on compression ratio, optimizing on latency of theencoding/decoding process, optimizing to subjective image quality basedon a model of the human vision system . . . . Decomposing theimage/video could mean splitting the image/video stream into rectangularareas, each area having a specific image type. For example: if thedesktop of a PC is to be replicated somewhere else then the desktopcontents will have to be transmitted over a channel and the desktop willbe displayed again at the receiving side. In case the channel isbandwidth-limited then typically encoding (compression) will beperformed before sending the data over the channel and decoding(decompression) the data at the receiver side. The typical situation isthat the desktop will consist of multiple areas: for instance there canbe an area containing a task bar, there can be an area containing a textdocument, there can be an area containing photographs, yet another areacan contain a video sequence and there can also be some part of thedesktop background visible. The present situation is that the desktopwill be compressed as a unit using some particular compression scheme.However, according to the present invention, at the transmitter side onewill detect that the desktop 100 as shown in FIG. 1 consists of severalimage types and for each of those areas a different compression schemecan be used. For the taskbar area 101 one could for instance use a lossycompression algorithm based on JPEG2000, for the text document 102 onecould use a simple run-length-encoding as this will compress text datawell at low complexity, for the photograph 103 one could use a losslessJPEG-LS scheme because one wants the photograph to be displayedcorrectly and for the video area 104 one could use a very complex MPEG-4video compression codec that achieves very high compression ratio butalso introduces high latency. The background area 105 can be heavilycompressed using JPEG or could even be replaced by a plain background tosave bandwidth. Detection of the image areas and according types can bedone automatically or with help from the application generating thecontents. In the first situation a classification algorithm will examinethe total area to be transmitted and decompose the area into one or moreobjects of specific image type. The classification algorithm can be forinstance a neural network classifier, a pattern recognition algorithm, ahistogram analysis algorithm, an algorithm that takes information fromthe operating system about window locations and types, . . . . It isalso possible that the classification algorithm takes the temporalcomponent into account. In other words: it is much easier to identify avideo/movie stream if one also compares consecutive frames instead ofonly examining the data within individual frames. For instance in FIG. 2the classification algorithm 210 discovers that the entire area 200actually consisted of 4 objects. A first object is a rectangular window201 that contains text and that has specific location and shape. In thisparticular situation location could be the coordinates of the upper leftcorner and the size could be height and width. However the presentinvention is in no way limited by the exact representation method. Thesecond object is a rectangular window of type photograph 202 at specificlocation x′ and with size z′. The third object 203 contains a videosequence. The object is of irregular shape 231 but it will berepresented as a rectangle 232 with specific location and shape. In thissituation also the pixels 233 that will contain padding will bedescribed, possibly by listing these pixels 233 or by describing thepixels as a collection of one or more other objects (for instance thepadding area consists of a number of rectangles with specific locationand size and triangles with specific location and size). In the same waya last object detected is of type background 204. The object also is ofirregular shape but for the ease of encoding it will be represented as arectangle with specific location and size. Again, also the paddingpixels will be described. The classification algorithm can work on aframe-by-frame basis or can take multiple frames as input. Takingmultiple frames as input makes it for example a lot easier to detectvideo sequences. As an alternative to only using a classificationalgorithm, the applications generating the images/video could give ahint to the encoding algorithm (or alternatively the user of theapplication could give this information to the classification algorithmor to the application that then passes it to the classificationalgorithm). In this case the application that generates the text windowfor instance can communicate to the classification algorithm that theapplication generates a window with image type text and that this windowhas rectangular shape with specific position and size. Of course it ispossible that the application communicates only part of this information(for instance only image type) or that it communicates more information(for instance desired compression algorithm, sensitivity to latency forthis application, maximum allowed latency, confidentiality level of thedata or desired encryption algorithm, priority level, desiredtransmission protocol, desired transmission medium, . . . ). It is alsopossible that only some of the applications communicate information tothe classification algorithm, for the other areas of the image/video tobe encoded the classification algorithm will then classify without thehints.

The object areas can be rectangular or could take any shape. It is alsopossible that one simplifies the detected area to a less complex shape.When supposing for instance that a circular area is detected containinga video sequence, and then it could allow for more simple compression(if the compression codec only accepts specific area shapes) if thatcircular area is extended to a rectangular area containing the circleand encoding that rectangular area. In that case padding of data can beinteresting: the pixels in the rectangular area that are not within thecircular area could be set to a constant value or any other specificpattern for the encoding purpose. It is also possible to overlaymultiple areas in this way, as shown in FIG. 3: suppose there area twoapplications, each having a rectangular window, each having a specificimage type, but one part of the first window is hidden with a part ofthe second window, then it could be interesting still to encode window Aas a unit, but where the part 301 of window A that is covered by windowB is replaced by padding pixels. Of course at the decoding side, oneshould replace the padding pixels in window A again with the pixels ofwindow B that are on top of window A. The advantage of this approach isthat since window A contains one type of image after replacing pixels ofwindow B, the encoding of window A as a unit will be efficient withoutthe need to split window A in multiple rectangular zones. Indeed, thealternative is to split window A into two rectangular areas 302, 303having the same type of image, and then also encoding window B as aunit. In case one simplifies the detected area to a less complex shapeone should be careful to avoid compression artifacts at the borders.Indeed: then one particular type of object can be encoded using twodifferent compression schemes because the object shape does not matchthe simplified detected area. In that situation border/blockingartifacts can be visible. To avoid this problem one could pickcompression algorithms with same characteristics (quality, type ofcompression artifacts, . . . ) so that the border will not be visible.

Based on the image types (also called object types) detected by theclassification algorithms and possibly based on information provided bythe applications that generate the images/video, the individual objectsthen are encoded with different image/video compression algorithms.These encoded data streams are then transmitted over the channel and atthe receiver side they are decoded again and the objects areregenerated. Finally the complete image/video is reconstructed based onthe composition of the individual objects where the effect of paddingpixels is removed (for instance in case of overlaying windows or objectsrepresented by other shapes than what the object in reality was). It isto be noted that the selected encoding algorithm could also compriseusing specific error detecting or correcting codes, using specificencryption algorithms, using specific transmission protocols (UDP or TCPfor instance), using specific transmission channels (for instanceselection between Bluetooth or Wifi or UWB), . . . .

It is to be noted that the “image type” can be very specific. In thecase of medical imaging for instance (but not limited to) one coulddistinct between “monochrome Chest Image”, “Mammogram”, “colour CTimage”, “colour ultrasound video sequence”, “rendered 3D object” . . . .The present invention is of course not limited to any specific imagetypes. Also an image type could refer to a particular type ofapplication (such as a PACS viewing application in medical imaging) oreven to one particular application (or even version of application) of aspecific vendor.

Of course decomposing the image/video stream to be transmitted is doneperiodically (can be done for each individual frame or only once everyfew frames) so that changes in object types can be detected and takeninto account. Alternatively the operating system or the applications cangive a hint to the classification algorithm if a change has occurred. Inother words: if the operating system creates, or changes state(position, size . . . ) of a window then it informs the classificationalgorithm so that a new object detection pass can be started. Also theapplication itself can inform the classification algorithm if itscontents type has changed.

According to another aspect of the present invention, the used encodingalgorithms can be adapted in real-time based on the current channelcharacteristics. Channel characteristics are for instance but notlimited to: currently available bandwidth, variance on availablebandwidth or statistics (for instance but not limited to min, max, mean,variance, distribution, time behaviour, . . . ) of available bandwidth,bit error statistics of the channel, statistics of latency introduced bythe transmission channel, . . . . This could mean for instance that ifthe available bandwidth of the channel decreases, that then some (forspecific object types) or all of the image/video compression codecs willbe reconfigured in order to have higher compression ratio or that othercodecs are selected for some or all object types. Alternatively if thenumber of bit errors introduced by the channel increases, then one coulduse better error detection codes or error correcting codes for one ormore object types that will require more bandwidth but will result inhigher quality at the receiver side. One could even measure (orestimate) the bit error rate at the receiver and then ask the sendingdevice to change the error correcting code so that at least a minimumquality level (bit error rate for instance) is obtained. Also the natureof the bit errors (burst errors, single bit errors . . . ) could be usedto automatically select the most suited error correction or errordetection code. Alternatively if the latency on the channel increases,then one could select compression algorithms with lower latency for oneor more object types in order to have lower overall latency (latency ofencoding/decoding and of the transmission channel). Alternatively if thevariance of the available bandwidth is rather high then one couldincrease the compression ratio for one or more object types in order toavoid temporary congestion of the network. In this case one couldincrease the compression ratio for object types that are considered tohave lower priority (such as text data) or for applications thatcommunicated “low priority”. One could also assign priority to specificdevices. For instance if a device is a portable display that receivesimages from a server through a wireless network and if there aremultiple devices, then it is interesting to also assign priority toparticular devices. For instance if in a hospital environment thosedevices are used both in the emergency room and also in regular doctor'soffices then it is obvious to give priority to the devices used in theemergency room if available network bandwidth is low. Giving priority toa device could mean in this case assigning more bandwidth tohigher-priority devices so that image quality is higher and userinteraction is more fluently or could mean reducing available bandwidthfor lower-priority devices or even disconnect (temporarily) the deviceswith lowest priority completely so that more bandwidth is available forhigh-priority devices. Deciding whether a device is high or low prioritycould be done by assigning priority to physical devices: for instance,devices intended for use in emergency room could have built-in higherpriority level. Alternatively it could mean deciding on priority basedon the location where the device is being used (for instance the samedevice has higher priority if used in the emergency room than if used ina doctor's office). Alternatively it could mean giving priority based onthe user that is using the device: for instance if a doctor is logged onto the device (or uses the device, this can be checked through password,fingerprint reader, iris scan, voice recognition, photo ID, . . . ) thenthis device will have higher priority than if a nurse was logged on tothis device (or uses the device). Determining who uses the device couldbe by means of a logon procedure with username and password or by meansof a fingerprint reader or by means of a security token, . . . .Alternatively priority of a device could be determined based on theapplications accessed through that device. For example: if a device isonly used at a particular moment in time to read email then this devicewill be assigned lower priority than same device when used to reviewmedical images. Of course any combinations of one or more methods todetermine priority as explained above are also possible.

It is to be noted that measuring the total latency of the system canbest be done by actually measuring the time between an action of theuser and the response to this action. For instance, when the userperforms an action (such as for instance but not limited to, a mousemovement, a keyboard command, a mouse click, . . . ) then a timer can bestarted and the time can be measured until the modified image that isresult of the user action is actually displayed at the client side. Thismeasuring of latency can be done for the multiple codecs that aresupported, for the available transmission channels, for the availableerror detection/correcting codes, . . . . In this way clear informationcan be obtained about total system latency. This information then can beused to configure the client-server system optimally. Another way ofmeasuring latency could be done by embedding signatures in the imagesthat are transmitted. In this case the client would send a command tothe server to start the latency measurement procedure. When the serverreceives this command then the server can embed a (possibly invisiblefor the user) bitpattern in the image. The client then can look for thisbit pattern. The time from sending the “measure latency” command to theserver to the time where the bitpattern was found is also an indicationfor the total latency of the system. This method is less accuratehowever, because the server performs an abnormal operation (embedding abit pattern) that is not necessarily a good indication for the latencyof a normal operation such as moving a window, . . . . Moreover, at theclient side the time when the bit pattern is detected is not necessarilythe time that the bit pattern becomes visible for the user. Althoughsomewhat less accurate, the second method still models the codec latencypretty good as the bitpattern is actually encoded and decoded by thecodec that is used. One has to be aware that the bit pattern can bealtered because of the compression/decompression, so care should betaken to make sure that the client can even detect the possibly alteredbit pattern.

According to another aspect of the present invention methods aredisclosed to increase security in case of portable devices. Withportable devices it is often the case that information displayed on thedevices is only intended to be used in a specific environment and is notto be distributed outside that environment. In a hospital environmentfor instance the HIPAA regulations protect the privacy of the patient bystating that medical data (including images) that can be linked to aspecific patient (for instance because the patient name or ID is shownin the data) should not be distributed unless it is necessary fortreatment or in the clinical interest of the patient (see the HIPAAregulations for more details). For example: a mammogram (breast X-ray)from a patient may only be shown on displays inside the radiologydepartment and not be distributed outside the department unless allpatient ID data is removed from the image. In the past this was easy toachieve by making sure that only the workstations in the radiologydepartment could access the server with the medical images. If an imagewas to be distributed outside the radiology department (for instance tobe reviewed by a colleague of another hospital) then all patent ID wasfirst removed before sending the image. However, with portable devicesand wireless networks the physical barrier between departments cannot beused anymore to protect patient privacy. Therefore, in one of itsembodiments, the present invention discloses a new solution. In eachdevice a method/system is included to determine the exact position ofthe device. This can be based for instance on GPS (outside in open air)or on RFID-tokens inside a building or any other method such as checkingwhich access point was used by the device. Yet another method to obtainposition is to check which access points can reach the device. Typicallyeach wireless access point can access (roughly) devices in a sphere witha specific distance as diameter. By checking which access points canaccess the specific wireless device and by measuring latency to theseaccess points one can easily limit the possible locations where theportable device can be. As a refinement one could replace the easy modelthat defines the range of an access point to real measurement datainstead of the easy “sphere” model. The exact method used to determinethe position is of course not relevant for the present invention. Basedon the exact position of the device the server will determine whether ornot a device is authorized to connect to the server and/or to receivethe specific data (for instance a mammogram with patient ID present)that was asked for. If authorization is granted then the servertransmits the data to the client over the (wireless) network butencrypted so that only that particular client is able to decode thedata. Of course it is possible to add extra conditions to giveauthorization to a client to see specific data. Examples are a login andpassword, a fingerprint scan, a particular device ID, . . . . To enhancesecurity it is possible to only transmit confidential informationthrough a wireless access point that is located inside the area wherethe data can be accessed. In that way a possible attacker will need tobe physically very close already to the confidential area before havingthe ability to attack the security system. Since the device is aportable device it is required to avoid that the confidential data isdistributed outside the allowed area. This can be done by automaticallyremoving the confidential data from the device if the device leaves theauthorized area. For instance: if a mammogram with patient ID data isshown on a portable display and the radiologists leaves the radiologydepartment then the device should automatically remove the confidentialdata from the display and also clear all cached instances (for instancein memory or on the hard-disk or other versatile and/or non-versatilestorage). Removing the confidential data could mean completely clearingthe display (and possibly clearing cached instances for instance inmemory or on the hard-disk or other versatile and/or non-versatilestorage) when leaving the authorized area. Alternatively, removingconfidential data could mean removing all confidential parts from thedisplayed image (and possibly clearing all cached instances for instancein memory or on the hard-disk or other versatile and/or non-versatilestorage) so that only non-confidential information is left visible whenleaving the authorized area. In this situation this would mean that theviewing application is still visible on the display but the confidentialimage is removed or replaced by an icon or any other symbol indicatingthat this information is not to be distributed and cannot be viewed now.Alternatively removing confidential data could mean related to HIPAAregulation making all medical images anonymous and removing all datathat could link a specific patient to the image displayed (and possiblyalso removing all cached instances for instance in memory or on thehard-disk or other versatile and/or non-versatile storage). In practicethat would mean that a mammogram that is viewed in the radiologydepartment and that has a patient ID visible, would still be visibleoutside the radiology department but the patient ID will automaticallybe removed when leaving the department. Alternatively removingconfidential data could also mean encrypting confidential data (andpossibly also encrypting all cached instances for instance in memory oron the hard-disk or other non-versatile storage) so that it is notaccessible anymore or using another encryption algorithm or changing theencryption algorithm strength. According to HIPAA for instance thiscould mean that inside the radiology department images are stored on thehard-disk of a portable device and encrypted using a particularencryption algorithm. When leaving the radiology department however thisdata will be re-encrypted using a stronger encryption algorithm tocomply with HIPAA regulations. It is obvious that the number ofconfidentiality levels is not a limitation for the present invention.For instance it is possible to have more than 2 security levels wherethe device would display less and less confidential data when thelocation where the device is used has lower and lower security level. Onthe other hand, the inverse can also take place: a portable deviceshowing limited data outside an authorised area, can automatically beadapted to show more data, e.g. patient's details, when entering theauthorised area. Again, this may be obtained by re-encrypting the datawith another encryption algorithm, or by doing the inverse of theanonimisation step, i.e. re-load the data for linking the images to apatient.

Yet another aspect of the present invention is complying with radiationregulations. If the client device has multiple transmission channelsavailable (such as for instance but not limited to Wifi, Bluetooth,Infrared, UWB, . . . ) then it can be that it is not allowed or notinteresting to use specific transmission devices in specific situations.For instance: in a hospital environment it can be that Wifi is onlyallowed in public places in the hospital while it is not allowed to useWifi in the intensive-care department (because of radiation power forinstance).

Another, non limitative example is shown in FIG. 4. In a hospitalenvironment, six security levels, being level 0, level 1, level 2, level3, level 4 and level 5 respectively are defined. Following definitionsof security level are given to these levels:

Level 5: this is the highest access level, e.g. radiology department:Data requested or transmitted to this level will contain all data.

Level 4: e.g. emergency rooms: Data requested or transmitted to thislevel will contain image data being made anonymous.

Level 3: e.g. intensive care room: Data requested or transmitted to thislevel will contain image data being made anonymous.

Level 2: e.g. the patient room: Data requested or transmitted to thislevel will contain encrypted image data.

Level 1: e.g. the intra-hospital rooms: Data requested or transmitted tothis level will only contain patient data.

Level 0: is region outside the hospital: all information will be removedfrom data requested or transmitted to this level, or all informationwill be encrypted.

According to the present invention the client device could autonomouslydecide which transmission channel to use based on the present locationand the regulations in place at that location. This could for instancemean that if a doctor uses a portable device that is currently using theWifi tranmission channel, and if the doctor goes into the operating roomarea, that then the device automatically switches from Wifi to forinstance infrared or DECT transmission mode in order to comply with theradiation regulations in place in the operating room area. It is to benoted that it is not necessary that the client switches to anothertransmission channel. The client could for instance also reduce theradiation power when this is required. With the GSM standard forinstance the client can emit radiation at multiple power strengths(measured in Watt), so reducing the emitted signal strength is also partof the present invention. The exact way how the client determines whereit presently is located and the exact way how the information aboutacceptable transmission methods at specific places is stores is not alimitation of the present invention.

As a non limitative example, a hospital environment is shows, in which 6zones are defined.

Zone 1: in this zone only GSM network is available.

Zone 2: in this zone, Wifi, DECT and UWB are available, next to the GSMnetwork. There is no restriction for use of any of the transmissionnetworks.

Zone 3: in this zone, Wifi and DECT are available, next to the GSMnetwork. For the GSM network, only transmitting with limited power isallowed.

Zone 4: In this zone, there is infrared (IR) transmission available,next to all transmission possibilities of zone 3. Only DECT and IR areallowed.

Zone 5: In this zone, there is Ethernet transmission available, next toall transmission possibilities of zone 3. Only IR and Ethernet areallowed.

Zone 6: in this zone, Wifi, DECT and GSM are available; there is norestriction for use of these networks.

Portable devices can only use transmission channels that are bothavailable at the portable device, and in the zone where the portabledevice is located, the portable device is only allowed to use theallowed channel in the zone where it is located. Possibly, the portabledevice will have to switch transmission channel if necessary.

Yet another aspect of the present invention is minimizing the requiredcalculation power of a client while still supporting multiplecompression protocols and algorithms. Since the present invention willuse different image/video compression algorithms for different parts ofthe image, it is required that the receiver side (client) can decode allthese image streams. However, since it is desired to select the bestcompression codec for each image type, it is to be expected that atremendous amount of compression algorithms must be supported both bythe server and the client. Moreover, the codecs used can change overtime based on channel properties resulting in even more codecs to besupported both at client and server side. At the server side this istypically not a problem as the server (that generates the images/video)typically is a powerful workstation with lots of memory, hard diskstorage and processing power. The client however is designed to be lightand portable and therefore has very limited processing power. Thepresent invention solves these conflicting requirements by disclosing aclient at the receiver side that can be reconfigured at multiple levels.One way to achieve this is by providing a reconfigurable device (such asa processor, a FPGA, a DSP, a GPU, a CPLD, an ASIC that can beconfigured to perform multiple operations, . . . ) or any combination ofsuch devices in the receiving client and to configure thesereconfigurable devices based on the compression codec that is currentlyrequired. In other words: if the server sends a JPEG2000 stream to thereceiving client then the client logic will be reconfigured in order tobe able to decode this stream. Reconfiguring the client can be done bymeans of configuration code that is already present in the clientdevice. One could for instance store configuration bitstreams for eachcodec that should be supported on a flash memory in the client deviceand the reconfigurable device (processor, FPGA, . . . ) would then beconfigured with the appropriate bitstream based on the currentlyrequired codecs. In this way it is not required to have support for allcodecs available all the time at the client side and this of coursereduces the amount of processing power and/or hardware resources that isrequired at the receiving client device. While this approach is usefulin an environment where a limited number of codecs will be used it soonbecomes impractical if a large amount of codecs must be supportedbecause the configuration code for all these codecs must be available onthe client device. Another limitation is that this method does not allowusing new codecs that were not available at production time of thereceiving client unless the client is updated. Therefore a bettersolution is to allow the client to be reconfigured completely or partlyfrom the server. This means that the server will send the configurationcode for the required codecs to the client making it possible to use anynew codec that is available at the server side. Of course at the clientside some logic must be present that makes it possible for the client toreconfigure itself after a new codec was sent to the client. This couldmean for instance that the initial configuration of the client allowsconnecting to a server, downloading some codec, reconfigures the clientfunctionality, and then connects to this or another server using thiscodec. Another alternative is that the client has a (limited) number ofcodecs available so that the client immediately is able to decode theserver stream (perhaps with non-optimal quality). This means of coursethat in the beginning the server should send data to the client in sucha format that the client can decode the data with the codecs that theclient has initially available. The client then starts to download thenew codecs that are needed for optimal performance while the client isalready functional. Once the required codecs are downloaded then theclient can be reconfigured and the server can use the new codecs. Theexact method of transferring codec configuration data does not matterfor the present invention. One possibility is to send the codec codethrough a separate connection between client and server. Alternativelyone could embed the new codec code into the bitstream of an existingchannel. This could mean for instance embedding the codec code intospare bits of a compressed video/image stream.

To avoid sending the same codec code over and over to the clients it ispossible that the client device locally saves a copy of the all ore somecodec code. In that way the client can communicate to the server that italready has available some codec and that there is no need to send thecode again. This also means that at the beginning of a session theclient can communicate a list of available codecs to the server. Theserver then can start streaming data to the client by using one or morecodecs out of this list and can also (if needed and supported by theclient) send new codecs to the client to further improve the efficiencyor quality of the transmission. Additionally the client could also senda quality-level or performance-level at the client side for codecssupported by the client. The server then can take this information intoaccount when deciding which codecs will be used for the transmission.For instance: a client could communicate to the server that the clientinitially has a JPEG-LS codec available, a raw-data transmission codecavailable, and a RLE-codec available. It additionally communicates tothe server that the client is able to reach 10 frames per second for theJPEG-LS codec at resolution 1024×768 and 8 frames per second for theJPEG-LS codec at resolution 1280×1024, less than 4 frames per second forthe raw-data codec with maximum resolution 1600×1200 and less then 6frames per second for the RLE-codec with maximum resolution 1600×1200.Furthermore the client communicates that can accept from the server theJPEG2000 (with specific version that is compatible with the clientplatform) codec and can achieve with maximum framerate of 2 frames persecond and maximum resolution 1280×1024. All this information then canbe used by the server to determine which are the best codecs to use forcommunication with this client. Of course the client can also send otheradditional information such as but not limited by: the resolutionsupported by the client (for instance 1280×1024), the colour modessupported (for instance 16 bit grayscale, or 3×8 bit colour or 256colour palette mode, . . . ), the frame rates supported by the clientfor different resolutions and colour modes, the type of networkconnection of the client, the battery level of the client (so that theserver can decide to switch to a less power consuming codec at theclient side), the list of preferred codecs of the client (codecs thatwill achieve high-performance and quality at the client side), the typeof hardware available at the client side and the codec-requirements tobe compatible with the client hardware, . . . . All this additionalinformation can be used by the server to determine which set of codecswill give best performance for the transmitted stream. As an alternativeto exchanging information about predicted performance of codecs, onecould also just test all or a number of codecs and measure actualperformance. In this case the server would send all or a number ofcodecs to the client. The client would try to configure the clientdevice with that codec and if succeeded the server sends one or a seriesof test images or video data to the client using this specific codecthat is tested. The client then tests if the codec has sufficientperformance (this can be in terms of image quality, speed, framerate,latency, . . . ) and sends back the results to the server. Then thisinformation can be used instead of estimates or predictions.

It is also possible to only reconfigure a limited part of the client.For instance if the client contains an FPGA then it is possible toconfigure a part of the FPGA to support some standard codecs and alsoreserve some part of the FPGA to implement some custom codecs if needed.In the same way these custom codecs could be stored in the client devicein (non-volatile) memory or could be retrieved from a server whenneeded. For instance a JPEG-LS codec could be implemented by default inthe FPGA and a JPEG2000 codec and a MPEG2 codec could be implementedtemporarily in the device because they are needed for decoding thecurrent data stream. Also this allows for parallel decoding of thestreams associated to the different object/image types as describedabove. Alternatively if the client device consists of an FPGA and aprocessor, then the client device could decide to (partly) configure theFPGA for acceleration of some (or all) parts of a codec that takes tomuch time to execute on the processor or achieves too low quality on theprocessor. For instance: implementing a filter with a lot of taps wouldtake a very long time to execute on a processor while it can be executedvery efficiently on an FPGA, a DSP or a GPU. The same principle holdsfor other types of devices, this means if a client has multiple devicesthen the client could decide to execute parts of or the entire codecalgorithm on specific devices based on performance or qualityresults/statistics or measurements.

According to another aspect of the present invention the typical latencyproblem will be reduced and hidden for the user. When there isinteraction between client and server latency often is a major problem.For example: an application runs on a server and the desktop of thatserver is transmitted and recreated at a client device connected to theserver using a wireless network. If the total latency between client andserver (this means encoding latency, transmission latency, decodinglatency, application latency at the server side, and feedback latencyfrom client to server) is very high then this will result in a systemthat is perceived by the user as being slow and non-responsive. Forexample: if the user at the client side would click a button then itcould take 20 ms to send that button-click message to the server. Theserver might need 10 ms to react to the button-click, the encoding of amodified image could take 20 ms, the transmission of the encoded imagecould take 10 ms, the decoding of the encoded image could take 30 ms anddisplaying the modified image at the client side could take 15 ms. Inthat case the total latency would be 105 ms meaning that only after 105ms the user would see the result of clicking the button. It is obviousthat a large latency will result into a slow and non-responsive system.The present invention solves this problem by reducing the latency and byhiding the latency for the user of the device. Reducing the latency inthe system can be achieved by splitting the data to be transmitted intolatency-critical and non-latency-critical data. If the data to betransmitted is a typical windows OS desktop with a spreadsheet, a videowindow, a taskbar and a background, then not all of these components areas critical to latency. Working in a spreadsheet for instance requireslow latency because the user very frequently interacts with the data andexpects rapid response. A video window where a movie is shown forinstance is totally not sensitive to latency. The user won't even noticeif the movie is actually delayed 2 seconds. Only in the beginning thevideo latency will be visible because the video stream will only startafter 2 seconds, but typically this is not considered to be a problem.Also the desktop background can tolerate large latency. A desktopbackground change is often considered as very low priority and almostnobody will care if it takes 5 seconds to see the effect of the change.In a bandwidth-limited channel it often happens that packets are queuedto be transmitted because the channel is operating at (almost) maximalbandwidth. In such a situation one can give priority to packets that arepart of a stream requiring low latency. Also for the object types (seeabove) that require low latency one could select a codec that has lowlatency compression and decompression. To know whether or not an objectis sensitive to latency one can detect the object type using theclassification algorithm described earlier or one could receive hintsfrom the application (or a window/part from the application) or from theuser of the system. Also if multiple transmission channels are available(for instance but not limited to Wifi, Bluetooth, Infrared connection .. . ) then one can select the low-latency channels for streams thatrequire low-latency.

As an alternative method to improve the latency problem the presentinvention discloses methods to hide the latency for the users of thesystem. This can be achieved by locally performing latency-criticaloperations on the client. One example is to locally generate the mousepointer. The mouse pointer is a very latency-critical object since it isextremely visible if the pointer lags behind the movements of the mouse.To solve the problem the mouse pointer can be generated locally on theclient instead of on the server. This means that if the user moves themouse then this message (“move mouse to coordinate [x,y]”) will becaptured by the client, and the mouse will be rendered at the newposition locally by the client. Of course the location change will alsobe forwarded to the server because it is possible that extra-sideeffects (such as change of mouse pointer shape or highlighting of textbelow the pointer or any other reactions from the application running atthe server because of the moved mouse pointer) would occur at theserver. Of course these side effects then are transmitted from server toclient and displayed at the client side. Locally rendering of thekeyboard is also interesting since it is very annoying if the symbolsthat are typed lag behind hitting the keys. Locally rendering thekeyboard would mean for example that in a text document the symbols arerendered and displayed directly at the client and again also forwardedto the server. Apart from keyboard & mouse it is also possible toperform other operations locally. One example is using locallookup-tables. In medical imaging for instance one very often useslookup-tables to perform window/level operations. Window/level is a kindof contrast enhancement by applying a lookup-table to an existing image.Typically the mouse is used to control the brightness and contrast ofthe image and therefore to change the contents of the lookup-table. Eachtime sending a new image from server to client when the window/levelsettings are changed is not only very bandwidth consuming, but also thewindow/level operations would be very slow due to the latency introducedbecause of transmitting images from server to client. Therefore a goodsolution is to apply the lookup-table locally at the client side. Thenthere are still two alternatives. One could send the mouse commands tothe server who then creates a new lookup table. This table can be sentto the client and the client then can locally apply the look-up table tothe image that is already present at the client. The second alternativeis that the client also locally processes the mouse command and locallycreates the new contents of the lookup-table. The image is then adaptedon the client side based on the newly created lookup table. Of coursethe mouse command is also sent to the server and the server will respondwith the correct lookup table. The client then checks if the locallygenerated lookup-table was correct and replaces if necessary its lookuptable by the new one received from the server and again processes theimage locally. It is to be noted that the present invention is notlimited to only one lookup table. It is possible to have multiplelookup-tables for different parts of the desktop or even to havedifferent lookup tables for different windows or applications. It isalso possible to have lookup-tables in parallel or in cascade (inseries).

An extension (a generalisation) to the approach of local lookup-tablesis using local filters. In this case the client will locally performoperations that can be handled efficiently at the client side if theseoperations can be clearly defined from the server side. In other words:if the client is capable of efficiently handling a specific operationlocally and if that operation can be described clearly by the server tothe client then it can be favourable to indeed handle this operationlocally for latency reasons. One example could be performing specificimage processing filters locally. Suppose that the same image is shownon the desktop in colour (define this as image A) and in grayscale(define this as image B). Then it can be interesting for the server tosend the definition of the transformation that transforms image A intoimage B to the client. This could be for example: B(x, y)=R[A(x,y)]*0.30+G[A(x, y)]*0.59+B[A(x, y)]*0.11. Or in other words: thisformula explains how image B can be created from image A by calculatinga pixel-by-pixel weighted sum of the red, green and blue subcomponentsof each colour pixel. This approach will decrease the required bandwidthbetween client and server and will also decrease latency as part of theoperations can be handled locally. Another example would be applying afilter on a part of an image and where this part of the image isselected by the mouse. Suppose a medical image is shown on the desktopand by means of the mouse the user can move a square over the image. Inthat square the image is filtered. In this situation the server couldsend the description of the filter to the client and the client couldthen locally apply the filter to the image. This will greatly decreasethe required bandwidth and also the perceived latency will besignificantly decreased because the filter is applied locally at theclient. It is to be noted that (part) of the reconfigurable logic of theclient can be perfectly used to perform such operations.

Yet another technique to hide latency for the user of the display is topredict parameters, send predicted parameters to the server and cacheresults from the server locally at the client so that the results arealready available when the parameters are actually valid. Suchparameters could include but are not limited to mouse commands(location, button operations, . . . ), keyboard commands, value ofcertain environment variables or program variables, or any combinationof such features, . . . . The exact method to predict these parametersis not relevant for the present invention. Some possibilities are usinginterpolation of mouse trajectories (predicting the next position of themouse by trying to fit the past mouse trajectory with a curve and thenpredicting the new mouse position by means of the fitted curve), justtrying out all or some parameter values in the neighborhood of thecurrent value, trying out a limited number of random values in theneighborhood of the present parameter values, . . . . For predictingparameter values one can use techniques from computer architecture.There techniques have been developed (“value prediction”) to predict thecontents of registers. As illustration an example is provided: if theuser makes a slow continuous movement with the mouse then the mostlikely the future positions of the mouse can be predicted with ratherhigh confidence. In this situation it is interesting to send thepredicted mouse locations to the server. The server then will respondwith a modified image based on the predicted mouse position. The clientshould then cache this image together with the mouse predicted positionthat corresponds to this image, the image itself is not displayed atthis moment. If the predicted mouse positions (or some of the predictedmouse positions) were correct (in other words if the user actually movesthe mouse to one or more of the predicted positions), then the clientcan immediately show the cached images corresponding to the correctpredicted mouse positions instead of having to wait for the response ofthe server. If the predicted mouse positions were not correct then theclient can do two things. The first possibility is to send the actualmouse position to the server and wait for the image corresponding tothis actual mouse position. The received image then can be displayed onthe client together with this correct mouse position. This approach willresult in higher latency and slower response but the displayed imagewill be completely correct. The second possibility is to show the mousepointer at the correct location on the client, but to display a cachedimage on the client corresponding to a mouse pointer location that islikely to be very similar to the actual mouse pointer position. In otherwords: suppose that the mouse pointer is actually at location (10, 10),but there is no cached image available for mouse pointer location (10,10). It could then be decided to use the cached image that correspondsto mouse pointer location (11, 11) because it is believed that thisimage will not differ a lot from the image that corresponds to mousepointer location (10, 10). The mouse pointer itself should of course beshown on location (10, 10) at the client because otherwise there will bediscontinuities in the mouse pointer movement. The decision to useapproach 1 (only use correct images) or approach 2 (still use cachedimages if parameters do not match completely) can depend on theparticular application being executed on the server, on the object typebeing shown, on the user logged in, on the available bandwidth of thechannel, on the preference of the user, or on any other condition. Theexact method used is not a limitation of the present invention.

The same principle of predicting parameters can be used for other thingssuch as but not limited to sliders, selection buttons, drop-down boxes,. . . . Also the same approach of predicting parameters is valid forhigher-dimensional parameter vectors. In other words: the image renderedby the server could be a combination of several parameters or parts ofparameters. For instance: the rendered image could depend on thecolumn-location of the mouse pointer and the status of a selectionbutton. It is even possible to actually find correlation usingstatistical methods between parameter or parameter vectors and renderedimages. Such an analysis could for instance make clear that the renderedimage is dependent only on the ratio between column position of themouse and the value of a parameter field. In that situation thepredicted values send to the server would be such that only combinationsof those parameters that would render different images will be sent tothe server, and also cached images can be used on the client if theratio of predicted column position of the mouse and the predicted valueof a parameter field are the same as the ratio between current columnposition of the mouse and the current value of a parameter field.

Of course, if a cached image corresponding to an earlier predictedparameter is used on the client, then still the current parameter issent to the server. This means that then the server will respond bysending the new rendered image corresponding to this new parametersituation. To further decrease required bandwidth the server could alsocache (some of) the combinations (parameters, rendered image) on theserver side. If the server then detects that the rendered image is thesame as the one that was sent earlier for the same parameter values,then the server can only send a message that it is the same imageinstead of sending the image itself. This reduces the requiredbandwidth. Alternatively, if the server does not cache the renderedimage then still the server could send a checksum (CRC) of the imageinstead of the image itself. Indeed, suppose the client sends apredicted parameter value and the server responds with a rendered image.Some time later the client actually needs the image for this parametervalue. The client uses the cached image but still sends the parametervalue to the server. Since the server did not cache the image the serverdoes not know that the image is the same as the image that was sent tothe client earlier (however, the server did keep track of previousparameter values so that the server knows that an image for thisparameter value was sent to the client earlier) the server sends a CRCof the image to the client. This CRC is only a very short messagerequiring very low bandwidth. The client now can calculate the CRC ofthe cached image for this parameter value to the CRC that the serversent to the server. If they match then there is a very good probabilitythat the cached image and the new image rendered by the server are thesame. In the other situation the client asks the client to send therendered image for this parameter value anyway because the imagesdiffer.

It is to be noted that the described method can also be used for anypart of the data to be transmitted from server to client. In otherwords: it could be that a parameter value only has influence on aparticular area on the display while the remaining area of the displayis uncorrelated or is correlated to other parameter values. This meansthat the method of sending predicted parameter values to the servercould be done in parallel for multiple areas. An obvious optimisation insuch a situation is to also send the desired area to the server. Ifindeed a particular parameter value only has influence on one particulararea of the total image, then it has no use that the server each timesends the complete image while only a part is correlated with theparameter value. Only sending the parts that the client is interested infurther reduces the required bandwidth.

Of course the method of sending predicted parameters and then usingcached images only works if a clear correlation can be found between theparameter value and the images. If for instance a movie is played andthe mouse movements have no effect on the movie then it also has no useto cache images together with mouse positions. There are several methodsto test whether or not the approach of predicting parameters and usingcached images makes sense for a specific situation. A first method is totry in advance if the parameter value is correlated with the image. Thiscan be done by sending multiple parameter values to the server, thenexamining the resulting images and verifying if exactly the same images(or sufficiently similar images) are regenerated if the same parametervalues are sent a second time to the server. This approach can berepeated if necessary to verify in time if the correlation betweenparameter values and images is still sufficient. A second method is tojust start using the system of predicted parameter values and cachedimages and checking if the system is useful when the images areverified. Indeed: first predicted parameter values are sent to theserver and the generated images are cached at the client side. If lateron the parameter values equal earlier predicted values, then the cachedimages can be compared (or the checksums of the images) with the newimages generated by the server corresponding to the same parametervalues. Based on the degree of similarity the approach of usingpredictions and cached images can be continued or stopped. The decisionwhether or not to use the prediction method can also be based on anyother condition such as preference of the user, type of applicationrunning at the server, type of image object, . . . . This exact decisionmethod does not limit the present invention. If the client concludesthat there is no or insufficient correlation between parameters sent tothe server and generated images by the server, then it is best to stopusing the prediction scheme to avoid risk of faulty states at the serverside. Indeed, if as a reaction to a predicted parameter the serverenters another not intended state then this was done unintentionally andcertainly not intended by the user. This results in confusion of theuser, as the new server state has to be displayed at the client side inorder to give the user the possibility to get back to the originalserver state. For instance if the client would send as predicted valueto the server the combination of “hold left mouse key” and “new mouseposition (x, y)” and as result of this action the server activatesanother window, then this state has to be shown on the client becauseotherwise the state of the server and the state that is shown on theclient would be inconsistent. By showing this actual state on the clientside the user can correct the problem by performing the necessaryactions to get the server back in the correct state.

It is in most situations relatively easy to link generation of a newimage on the server side to a parameter value sent by the client. Inmany situations the image generated by the server will be static (or atleast part of the image will be static) and will change shortly afterthe client sent a parameter value to the server. The client then canassume that the change of image was due to the parameter value sent tothe server. A more reliable system is that the server adds a message tothe generated image that links the new image to a particular action ofthe client. For instance, the server could add as metadata to an imagethat the image was the result of sending parameter value X from clientto server. An additional refinement could be that the server postponessending a new rendered image to the client until the image content ofthe server again reached a stable situation (after sending a parametervalue from client to server), or until a maximum time has passed. Forexample: if the client sends a mouse location update to the server, andthis mouse location update results in a small animation at the serverside and then results in a new stable condition, then the server couldwait for the stable condition to send an image update to the client. Theadvantage is that correlation between parameter value and image contentswill be much more likely and that required bandwidth is further reduced.Furthermore it might be likely that indeed the animation is indeed atransition situation that is of no or low interest for the user so thatnot sending these information/images to the client will not pose a majorproblem. The disadvantage is that indeed the animation will not bevisible for the user of the client. The server could additionally labelthe images sent to the client with the time that was needed to reach thestable condition after the parameter value sent from client to serverwas processed. Also if no stable condition was obtained then the servercould send this as metadata to the client, the client then can make useof this information, for instance by concluding that this parameter orset of parameters might not be correlated with image contents that wereexamined. Also the client could inform the server whether a parametervalue sent to the server is a predicted value or an actual value. If itis a predicted value then the server could indeed wait for a stablecondition while if an actual value it might be more interesting toimmediately process the parameter update. Also if the server knows thata parameter sent to the server is a predicted value, then the server cansave its local state. Indeed: if the parameter value is a predictionthen there is a possibility that if the prediction turns out to bewrong, the server arrived in a wrong state because of using thepredicted parameter value. If the server first caches its state beforeusing the predicted parameter value, then it is always possible toreturn to the correct state if the prediction turned out to be wrong, oreven by default go back to the state before the prediction was used. Thestate of the server comprises all information that is required to undothe effects of using a predicted value. For example: the client informsthe server that a prediction of the mouse position needs to beprocessed. The server first saves it state (this could include but isnot limited to: window attributes such as in front/back, minimized,maximized, size, position, important program variable values, . . . )and then executes the “change mouse position” command locally. Theserver then sends back the result (rendered image) to the client. Thenthe server restores its state so that the server remains in the correctstate if later on the predicted mouse position turns out to be wrong.This by default restoring of the state is in theory not always required.If using the predicted parameter value does not change the state then ofcourse it has no use restoring the state. However, suppose that becauseof using the predicted mouse position another window is activated, andthen it is absolutely required to restore the correct state beforeprocessing any other predicted parameter values from the client. Indeed,because using a predicted parameter value can result into another serverstate it is also possible that because of this the reaction of theserver to a new predicted parameter value sent by the server will bedifferent compared to the situation where the first predicted parametervalue would not have been executed by the server.

It is to be noted that in most situations it will not be a problem ifthere is a small error on the predicted parameter value. In other words:in most situations it will not be a problem to use a predicted parametervalue that is not the same but is close to an available predictedparameter value. This of course can be dependent from application toapplication. Whether one uses these incorrect predicted parameter valuescan for example be made dependent on the type of application or can evenbe decided on basis of a list of individual applications. Of course theapplication could also communicate to the encoding algorithm whether ornot using wrong predicted parameter values is acceptable or not.Alternatively the user could select this and give this information tothe encoding algorithm, possibly through the application.

According to another aspect of the present invention a series of cachedimages will be stored at the client side and optionally also at theserver side. When the server sends an image to the client then theclient could decide to cache that image locally if the client believesthat this image will need to be transmitted in the future again. At theserver side the server could also cache this image or could cache achecksum (for instance CRC or hash) on this image. If the server then inthe future finds out that an image needs to be sent to the client andthat this image was sent before to the client, then the server can firstsend the CRC or hash value of this image. The client then can checkwhether or not this image is available at the client side. This can bedone by verifying the CRC or hash value that was sent by the server tothe client. If the image was indeed cached at the client side then theclient can immediately use this cached image so there is no more need tosend the entire image over the transmission channel and this will ofcourse reduce required bandwidth and reduce latency. If the CRC sent tothe client does not equal any CRC of the images available in the clientcache then the client has to send a request to the server in order tohave the server transmitted the image anyway. At the server side theserver can decide to store the entire image in the cache or only storethe hash or CRC of this image in the cache. Indeed, to verify if thepresent image was sent already before the server can compare completelythe present image with the images in the cache of the server, oralternatively the server can calculate the hash or CRC of the presentimage and compare this hash or CRC with the hash or CRC values isavailable in the cache of the server. If the hash or CRC values of thepresent image to be transmitted match (equal) with the hash or CRC valueof a previously transmitted image then the probability is extremely highthat the images themselves also match. Only storing the hash or CRCvalue furthermore reduces calculation power at the server side becauseinstead of comparing all images in the cache of the server with thepresent image, only the shorter hash or CRC values of the images have tobe compared. Of course also required cache memory/storage is reduced. Toreduce required calculation power at the client side, the server couldby default also send the calculated hash/CRC value of the image that isbeing sent to the client together with this image. This removes the needfor the client to calculate this hash/CRC value. The client then canimmediately store the image in the cache together with the hash/CRCvalue that was calculated by the server.

Of course there is no need to store all images in the cache that aresent to the client. The client could decide to only store some images inthe cache together with the according hash/CRC value. Alternatively theserver could decide which images should be cached. This can becommunicated to the client for instance by only sending a hash/CRC valuetogether with the images that should be cached at the client side. Theserver could also explicitly send a command to the client to cache aparticular image. Deciding which images need to be cached can forinstance be done by just looking how many times a specific imageoccurred in the past. If this number exceeds a specific threshold valuethen the server/client could decide to cache this image for future use.Alternatively the application could give hints about key images to theserver/encoding algorithm. These key images then can be cached. Ofcourse any combination of the explained strategies for deciding whichimages should be cached are also possible.

Instead of using CRC/hash values to identify specific images the servercould also just give a unique identifier to the images. One example is asimple numbering scheme. The server could for instance label the firstimage that should be cached by the client with number 1, the secondimage to be cached by number 2, and so on. If an image occurs again thatwas already sent before and was cached then the server can just sendthis unique identifier to the client. The exact method of uniquelylabelling the images of course does not limit the present invention.

The server should also have the possibility to clear one or more entriesin the client/server cache. For instance if the cache at client/serverside is only 256 images in size then the server could send a clear cachecommand to the client once the cache is full and a new image needs to becached. This command will result in clearing the cache at both theclient and server side so that new images can be cached in the future.Alternatively the cache clearing can take place in a round-robin scheme.In other words: if 256 entries out of 256 entries are used then thefirst entry (number one) would be overwritten if a new image were to becached. If another new image is to be cached then the next entry (entrytwo) would be overwritten. Alternatively the server/client could decidewhich cached images can be removed. For instance: if the server has animage cache of 256 images but the client has only an image cache of 16images then the client could select cached images that are to be removedif a new image is to be cached based on a frequently-used statistic. Inother words: the client will remove images from its image cache based onhow much these images were used in the past. If the cache is full thenthe client will first remove images that were not/not often used in thepast. In case of a server cache of 256 images and a client cache of 16images this could mean that the pairs (image, CRC) at the client sideare extended with a third value named “usage”, that indicates how manytimes this cached image was used until now. If an entry has to bedeleted because the cache is full and a new image has to be cached thenthe client will first remove the cache entry with lowest “usage” value.Alternatively, to reduce calculation power at the client side, theserver could manage the cache at the client side completely orpartially. For instance: the client could keep the usage statistic orcould keep track of the cache status of the client side and decide basedon this information what the best strategy is for the client cache. Theserver in this case will control the client cache or would give hints tothe client cache. A very useful application can be in medical imaging.In CT reconstructed volumes the user typically browses through a largenumber of slices. This is done by pulling a slider with the mouse. Theradiologist typically browses forward and backward a lot through thelarge number of slices. In this situation it is very likely that imagesthat are displayed will be displayed again in the future. Indeed,suppose the radiologist looks at image 1, 2, 3, 4, 5, 6, and then goesbackward again to 5, 4, 3, 2. Then these images can be reused from thecache. The simplest way of implementing this would be that the serverjust numbers the images and sends the images together with the number tothe client. If an image has to be displayed that was sent in the pastthen the server can send the unique ID (number) instead of the image. Itwould even be better if the application itself can give hints to theencoding algorithm about how the images should be numbered. Indeed, theapplication itself that generates the images (slices from the volume)has the most knowledge about what slices are viewed and which slices arelikely to be viewed in the future again. Another way of viewing thistype of data (browsing through slices of reconstructed CT volumes) is byjust looping through the slices. This means that the user hits a kind ofplay key and then the slices are shown in order, so slice 1, slice 2, .. . , the last slice and then the browsing automatically restarts againwith slice 1, slice 2, . . . and so on. In this situation theapplication perfectly knows which images will be needed in the future.Therefore the application itself can also predict a parameter (slicenumber in this case), the server can generate the image for thepredicted parameter (slice number) and it can already be sent as acombination (predicted parameter, image) to the client. Alternatively,the images corresponding to images that will be likely to be shown inthe future can also be sent to the client as being images that should bestored in the cache but only if the client does not show/display theseimages immediately but just places them in the cache for possibly futureuse. This would mean in the case of going through a series of slicesthat the server would render the images (slice 1, slice 2, slice 3,slice 4) immediately and send these images to the client as being imagescorresponding to predicted parameters or as images to be caches. Theimages will then already be available at the client side by the time theimages need to be displayed at the client side. To be more specific:suppose that the radiologist looks at the slices (images) in order(image 1, image 2, image 3, . . . ) at a rate of 2 images/second. Thenthe server could already send as much images as possible to the client(for instance 4 images/sec, namely image1, image 2, image 3, image 4, .. . ) so that all these images already available at the client side bythe time the client needs to display these images. This will reduce invery low latency (the images are available on-time) and increasedperceived reaction-speed of the client-server architecture. Of coursethis system also works if the radiologist browses himself through theslices and possibly goes forward and backwards through the images (forinstance by pulling a slider). If the server generates images in advance(possibly based on hints by the application) and sends them already tothe client to be cached, then the images will be available on time evenif the image viewing sequence is not completely regular.

It is a goal of the present invention to make it possible to replicatethe desktop of a server anywhere else by connecting to it from a(wireless, portable) client device. The starting point is that allapplications should be running at the server end and this to limit themaintenance cost of the client/server architecture. Indeed, if allapplications are installed and run on the server then only the serverneeds to be updated if an application update is required. Also if betterhardware/processing capabilities become available (such as faster GPUsor CPUs) then only the server needs to be updated and the performance ofall the clients will increase as well. Therefore the conceptual designinvolves replicating the desktop contents of the server to a remoteclient display and allowing feedback from the client so that for theuser of the client it appears as if the user was working locally on theserver. In practice this could be implemented by using “screen-scraper”technology that just grabs the screen content of the server at anymoment and encodes and transmits this screen contents to the client. Theclient then decodes this encoded data stream and regenerates the imagesat the client side. Any inputs from the user at the client side are sentto the server and the server (application(s) running on the server)processes these inputs. To allow the user to interact with the server atleast a keyboard and mouse connection should be available. This can bedone for instance by converting the analogue PS/2 signals of mouse andkeyboard at the client side into digitally sampled signals andtransmitting these digital samples to the server side. At the serverside the sampled PS/2 signals are reconverted into an analogue signalagain and are connected with the PS/2 input of the server. In this waythe system will behave as if the keyboard and mouse were directlyconnected to the server. An alternative is that the keyboard and mouseat the client side are also processed locally in order to reducelatency. These techniques have been described above. It is also possibleto support a USB connection at the client side. Here the problem issomewhat more difficult because the USB protocol requires strict timingsand converting the USB signals to digital samples and reconstructing theanalogue signal at the server side could take too much time and resultinto protocol problems. Also the USB is a bi-directional protocol. Toovercome this problem solutions exist such as the “extreme USB” solutionthat handles a part of the USB protocol locally to overcome the stricttiming requirements of the USB protocol.

Of course it should also be possible to connect with multiple clients tothe same server. This can be done by using a multi-user operating system(such as Windows 2000 server or linux) where multiple user sessions cantake place in parallel. In this case for each user that wants to connectto the server a session (having its own framebuffer) can be created.Each of these sessions then can be treated as a desktop that must betransmitted to a particular client. Another possibility in case there isno multiple-user support from the operating system is to create virtualwindows. This could mean for example that for each client a window iscreated with the application that the client wants to run. All theseapplications however run in one user space of the single-user operatingsystem. However, only the windows belonging to each specific client aretransmitted to that client. For example: if a server is intended to runa specific application that requires high processing power, and it isthe intention that multiple users can use that application at the sametime. Then for each user an instance of that application would bestarted and only the windows belonging to the instance of each specificuser will be sent to that specific user. Alternatively, one couldprovide support for multiple users in the application itself. In thatcase the application would handle serving multiple users at the sametime and only the windows/parts of the application that belong to thesession of a particular user would be sent to that particular user.

Of course, all aspects of the invention that are described earlier inthis text can be applied in this particular implementation. For instancebut not limited to: locally handling some parts such as keyboard ormouse, locally executing some parts of the application to reducelatency, using the mechanism of predicting parameters and using imagesassociated with these predicted parameters, caching images locally atclient/server and uniquely identifying these images and sending theidentifier instead of the image itself if the image must be sent again,. . . .

1-33. (canceled)
 34. A method for securing transmission of data from aserver to a portable imaging device, the data comprising confidentialand non-confidential data, the method comprising: determining a positionof the portable imaging device with respect to an authorised area, basedon the determined position of the portable imaging device, determiningwhether the portable imaging device is authorized to receive theconfidential and/or non-confidential data over a predeterminedtransmission channel, transmitting, from the server to the portableimaging device, the confidential and/or non-confidential data requestedif authorisation is granted, the portable imaging device having adisplay area, removing at least from the display area at least theconfidential data when the portable imaging device leaves the authorisedarea, so that only non-confidential information is visible, and showingat least on the display area the confidential data when the portableimaging device enters the authorised area.
 35. The method according toclaim 34, further comprising removing at least confidential data fromvolatile and/or non-volatile memory elements in the portable imagingdevice when the portable imaging device leaves the authorised area. 36.The method according to claim 34, furthermore comprising encrypting theconfidential data when the portable imaging device leaves the authorisedarea.
 37. The method according to claim 36, the method furthermorecomprising decrypting the confidential data when the portable imagingdevice enters the authorised area.
 38. The method according to claim 34,the method furthermore comprising switching to a transmission channeldifferent from the predetermined for transmitting the requested data,the transmission channel used depending on the determined position ofthe portable imaging device.
 39. The method according to claim 34, themethod furthermore comprising switching to a transmission channeldifferent from the predetermined for transmitting the requested data,the transmission channel used depending on regulations in place at thedetermined position of the portable imaging device.
 40. The methodaccording to claim 38, the portable imaging device determining whichtransmission channel to use for transmitting the requested data.
 41. Themethod according to claim 39, the portable imaging device determiningwhich transmission channel to use for transmitting the requested data.42. The method according to claim 38, wherein the server determineswhich transmission channel to use for transmitting the requested data.43. The method according to claim 39, wherein the server determineswhich transmission channel to use for transmitting the requested data.44. The method according to claim 34 wherein the confidential andnon-confidential data are images and/or video.
 45. The method accordingto claim 44, wherein the images and/or video are medical images and/orvideo.
 46. The method according to claim 44, wherein the transmissionchannel for transmitting images and/or video is a bandwidth limitedtransmission channel having varying available bandwidth for transmittingto a plurality of portable imaging devices, the method comprising theuse of a classification algorithm for each of the images and/or video,for: decomposing the images and/or video to be transmitted into multiplespatial areas, each area having a specific image type; detecting theimage type of each of those areas separately selecting for each of thoseareas an image and/or video encoding algorithm having a compressionratio; wherein each of the portable imaging devices is prioritized, saidclassification algorithm increasing the compression ratio of the imageand/or video encoding algorithms dedicated to a device having lowerpriority in case of decreasing bandwidth.
 47. The method according toclaim 46, wherein said prioritizing of the portable imaging devices isdone based on the applications accessed through said portable imagingdevices.
 48. The method according to claim 46, wherein said prioritizingof the portable imaging devices is done based on the identity of usersusing said portable imaging devices.
 49. The method according to claim34, further comprising a step of user log-on to one of said portableimaging devices.
 50. The method according to claim 46, wherein saidprioritizing of the portable imaging devices is done based on thelocation of said portable imaging devices.